Digital technology has become the dominate form of imaging for scientific, media and entertainment applications. Charged coupled device (CCD) and complementary metal oxide semiconductor (CMOS) sensors have been available since the 1970's; however due to limitations in pixel size, color depth, and spatial resolution, it has been necessary to employ creative means to improve resolution. Multiple efforts have been made to eliminate noise and to smooth edges. Along with curve fitting algorithms which involve mathematical abstraction, the visualization of an image was improved. Techniques for sub-pixelization were an important step in resolution improvement, especially for common technologies such as high definition television and video.
U.S. Pat. No. 5,251,037, filed in 1992, describes the ability to take advantage of microprocessors in order to process multiple images from a CCD. This method achieved high resolution by controlling exposure while moving a camera with an isolated CCD.
Work described in the paper “High Resolution Image Reconstruction From Lower-Resolution Image Sequences and Space-Varying Image Restoration,” published in 1992, by Telkap et al., demonstrates how the frequency of pixel values can be evaluated and noise can be reduced by using a Fourier transform. While this was an important step in the realization of resolution improvement, without the help of higher density CCDs, the interpolation improved images but not overall resolution.
A technical paper entitled “IEEE Transaction on Image Processing, VOL. 5, NO. 6, June 1996” further describes this type of improvement. A process by which pixels are blurred in order to provide a smooth curve, and avoid aliasing, aided digital video in becoming more spatially appealing.
Methods have also been proposed for taking various images, such as the 30 frames per second used in film, and most video, and filling in the unknown area between pixels by employing mathematical means in order to obtain an image which is superior to that defined by the screen, or camera specifications. While this is sometimes called sub-pixelization, it does not directly provide a method by which to observe objects of sizes smaller than one pixel, but instead either extrapolates from trends or overlays known values.
An example of sub-pixelization is described in U.S. Published Patent Applic. No. 2007/0171284 A1. A specimen is focused on an imager, and multiple images are captured, in succession, with the specimen being moved relative to the imager by a distance that is less than the size of a pixel. Whenever a shift in color is observed (that is a change in one of the red, blue or green values) an averaging method is employed to gain a smooth transition between pixels.
Prior art for sub-pixelization has been described through the precise decision of movement direction, which can slice a pixel in a way which allows for edge detection and interpolation to create super resolution images. Through digital treatment techniques such as curve fitting, dithering and simulation, pixel values can be assigned for sub pixels, rather than just the larger pixels of the captured image. An example from patent literature can be seen in publication US 2009/0028464. As with the previous example, the sub-pixelization produces higher resolution images due to an increased gradient.
U.S. Published Patent Applic. No. 2006/0133641 discloses a method for obtaining sub-pixel resolution by evaluating changes in intensity throughout movement. This is similar to the above described patent, but takes in account z direction movement (movement toward or away from the imager). The invention utilizes averaging in three dimensions, in order to create a superior two dimensional final image.
The known high resolution extrapolation techniques are well described by Borman in “Super Resolution for Image Sequences—A Review” in Proceedings of the 1998 Midwest Symposium on Circuits and Systems, 5 Apr. 1998. This in-depth paper described various methods employing various statistical functions to achieve improved final images. Markov chains, and other complex techniques are successfully described, however there is no method provided to either directly track individual sub-pixels or directly solving for sub-pixel values.
In the field of nano-microscopy Putman, U.S. Patent Applic. No. 2009/0028463, describes a method by which sub-pixel locations are defined and mapped through piezo electric nanomovement of a specimen relative to an imager. By moving at known distances smaller than a given pixel, the location of individual sub-pixels can be established and recorded. If a defined location is followed, a statistical function, such as the median, maximum or minimum value, can be taken, and sub-pixel resolution achieved. This method still requires statistical approximations rather than directly solve for a sub-pixel, which the present patent will describe.
All of this is possible, and has been explored in the patent literature because of the ever increasing number of pixels on a sensor, as well as increased processor speeds, allowing for sharper, clearer images on both the micro and macro scale. Piezo-electric translation stages have made it possible to move at distances smaller than a pixel. This innovation makes it possible to continue to sub-pixelate even as pixel size and density increases on CCD and CMOS sensors. Unique algorithms which analyze and reconstruct images make digital still and video photography, through sub-pixelization, more efficient.
A method is needed to provide sub-pixelization and image reconstruction directly and rapidly, without the need for mathematical approximation.